Devices, systems, and methods for anchor-point-enabled multi-scale subfield alignment

ABSTRACT

Devices, systems, and methods obtain a reference image; obtain a test image; globally align the test image to the reference image; select subfields in the test image; align the subfields in the test image with respective areas in the reference image; warp the test image based on the aligning of the subfields; select anchor points in the reference image; select anchor-edge points in the reference image; realign the subfields in the warped test image with respective areas in the reference image based on the anchor points in the reference image and on the anchor-edge points in the reference image; and warp the warped test image based on the realigning of the subfields.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.16/248,480, which was filed on Jan. 15, 2019 and which claims thebenefit of U.S. Application No. 62/618,840, which was filed on Jan. 18,2018, and the benefit of U.S. Application No. 62/758,275, which wasfiled on Nov. 9, 2018.

BACKGROUND Technical Field

This application generally concerns image alignment.

Background

Some computer-vision systems align two images. For example, somenondestructive testing systems, which examine the properties of objectswithout causing damage to the objects, align images of objects beforesearching for object defects in one or more of the images. Thesetechniques can be used in a quality-control process to identify defectsin objects.

SUMMARY

Some embodiments obtain a test image; globally align the test image tothe reference image; select subfields in the test image; align thesubfields in the test image with respective areas in the referenceimage; warp the test image based on the aligning of the subfields;select anchor points in the reference image; select anchor-edge pointsin the reference image; realign the subfields in the warped test imagewith respective areas in the reference image based on the anchor pointsin the reference image, the anchor-edge points in the reference image,and warp the warped test image based on the realigning of the subfields.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example embodiment of an image-alignment system.

FIG. 2 illustrates an example embodiment of a rigid alignment that showssome limitations of rigid alignments.

FIG. 3 illustrates an example embodiment of an operational flow foraligning images.

FIG. 4 illustrates an example embodiment of image preprocessing.

FIG. 5 illustrates an example embodiment of global alignment.

FIG. 6 illustrates an example embodiment of an image that has been splitinto subfields.

FIG. 7 illustrates an example of an image subfield that was alignedusing cross correlation.

FIG. 8 illustrates the results of an example embodiment of multi-scalesubfield (MSS) alignment.

FIG. 9 illustrates the results of an example embodiment of MSSalignment.

FIG. 10 illustrates the results of an example embodiment of MSSalignment that shows some limitations of MSS alignment.

FIG. 11 illustrates example embodiments of anchor points in an imagesubfield.

FIG. 12 illustrates example embodiments of anchor points in a completeimage.

FIG. 13 illustrates example embodiments of anchor-edge points.

FIG. 14 illustrates example embodiments of the differences between areference image and an MSS-aligned image and the differences between areference image and an anchor-point multi-scale-subfield-aligned(APMSS-aligned) image.

FIG. 15A illustrates an example embodiment of pixel shifts.

FIG. 15B illustrates an example embodiment of pixel interpolation.

FIG. 16 illustrates an example embodiment of a mapping of a pixel froman APMSS-aligned input image to an unaligned input image.

FIG. 17 illustrates an example embodiment of an operational flow foraligning images.

FIG. 18 illustrates an example embodiment of the discreteauto-correlation for two different points in an image.

FIG. 19 illustrates an example embodiment of the results of anchor-pointdetection.

FIG. 20 illustrates example embodiments of alignability maps fordifferent window sizes.

FIG. 21 illustrates an example embodiment of an operational flow toobtain anchor points from alignability maps.

FIG. 22 illustrates example embodiments of anchor points.

FIG. 23 illustrates example embodiments of anchor points and anchor-edgepoints.

FIG. 24 illustrates an example embodiment of a circular weighted maskthat may be used for cross correlation.

FIG. 25A illustrates example embodiments of a reference-image subfieldand an input-image subfield.

FIG. 25B illustrates example embodiments of the circular central areasof a reference-image subfield and an input-image subfield.

FIG. 25C illustrates example embodiments of a reference-image subfieldand the boundary (edge) areas of an input-image subfield.

FIG. 26 illustrates an example embodiment of a circular weighted mask.

FIG. 27 illustrates an example embodiment of a circular weighted mask.

FIG. 28 illustrates an example embodiment of an image-alignment system.

DESCRIPTION

The following paragraphs describe certain explanatory embodiments. Otherembodiments may include alternatives, equivalents, and modifications.Additionally, the explanatory embodiments may include several novelfeatures, and a particular feature may not be essential to someembodiments of the devices, systems, and methods that are describedherein.

FIG. 1 illustrates an example embodiment of an image-alignment system.The system 10 includes one or more image-alignment devices 100, whichare specially-configured computing devices; one or more image-capturingdevices, such as an x-ray detector 110A or a camera 110B; and at leastone display device 120.

The one or more image-alignment devices 100 are configured to obtain oneor more images of an object from one or both of the image-capturingdevices. The one or more image-alignment devices 100 are also configuredto align the one or more images of the object with a reference image,which may be an image of a defect-free object.

A high-precision alignment allows a good comparison of a reference imagewith another image of an object (e.g., an image of an object that isbeing tested). To allow a pixel-to-pixel comparison of the images, theone or more image-alignment devices 100 may align the other image withthe reference image at a pixel level or a sub-pixel level.

Also, in some circumstances a general rigid alignment will not work. Forexample, when an X-ray image is a two-dimensional image of athree-dimensional object, the two-dimensional image shows a flattenedimage of the object's three-dimensional shape or structure (e.g., atwo-dimensional projection of the three-dimensional object). Anyvariation in depth, rotation, or orientation in the three-dimensionalpositions may make a rigid alignment incapable of handling thedistortion caused by these three-dimensional variations. Also, a generalnon-rigid alignment may cause problems, for example when a goal of thealignment is to make the geometry of the three-dimensional objectconsistent in all the images, because the geometry could be altered by ageneral non-rigid alignment.

Thus, some embodiments of the one or more image-alignment devices 100perform an alignment that combines intensity-based alignment,feature-based alignment, rigid alignment, and non-rigid alignment.

FIG. 2 illustrates an example embodiment of a rigid alignment that showssome limitations of rigid alignments. A rigid alignment that is based onthe shifts on the x and y axes, on the rotation angle, or on theparameters of a Nomography transformation is generally adequate togenerate a rough alignment between two images. However, a rigidalignment may not be adequate in situations that require asub-pixel-level alignment. This inadequacy can be illustrated using thevariations of the distances between three selected positions in eachimage across a total of fifty images, which were taken from differentmachined parts that were built using the same CAD model, as shown inFIG. 2.

The top-left of FIG. 2 shows three positions: A, B, and C. The positionsA, B, and C were selected in each of the fifty images, and the positionsare landmarks that can be used to align the images. The distancesbetween the three pairs of positions (i.e., the distances between A andB, B and C, and C and A) were measured. The three distance measurementsfrom the image at index 0 are used as the point of reference (distance0) in this example. The distance differences of all other images againstthe image at index 0 are shown in the three other sub-figures. Thetop-right shows the differences of the distances from A to B, thebottom-left shows the differences of the distances from B to C, and thebottom-right shows the differences of the distances from C to A.

If the transformation for aligning different images can use a parametricapproach (e.g., a rigid alignment), significant variations between themeasurements of the three distances across different images should notbe present. However, significant variations are present. For example,compared to the other images, image 15 has significant differences fromimage 0 in all three measurements (A to B, B to C, and C to A), butimage 43 shows a significant difference for only one measurement (C toA).

Moreover, a general non-rigid alignment could produce unwanted results.For example, although a non-rigid alignment might produce only a smallcomputational error, it may distort the structural information.

Accordingly, some embodiments of the one or more image-alignment devices100 perform the following operations: They split the image into multiplesubfields (e.g., patches, areas). Then they align each subfield of oneimage to the corresponding subfield in one or more other images, andthey generate shifts in the x and y directions for each subfield basedon the alignment. The shifts may be interpolated and converted intorespective shifts of each pixel. Finally, they warp the image based onthe pixel shifts in the x and y directions to generate an aligned image.

Also, some embodiments of the one or more image-alignment devicesperform a global alignment on an image to get a rough global shift forthe whole image in the x and y directions. Then these embodimentsperform a multi-scale subfield (MSS) alignment to get a sub-pixel-levelalignment for each subfield. After that, these embodiments perform ananchor-point multi-scale-subfield alignment (APMSS) and use its resultsto finely adjust (e.g., shift, warp) the image to produce an alignedimage with structure information that is undistorted.

FIG. 3 illustrates an example embodiment of an operational flow foraligning images. Although this operational flow is presented in acertain order, some embodiments may perform at least some of theoperations in different orders than the presented order. Examples ofdifferent orders include concurrent, parallel, overlapping, reordered,simultaneous, incremental, and interleaved orders. Thus, otherembodiments of the operational flow may omit blocks, add blocks, changethe order of the blocks, combine blocks, or divide blocks into moreblocks.

Furthermore, although the embodiments of this operational flow and theother operational flows that are described herein are performed by animage-alignment device, some embodiments of these operational flows areperformed by two or more image-alignment devices or by one or more otherspecially-configured computing devices.

The operational flow in FIG. 3 starts in block B300 and then splits intoa first flow and a second flow. The first flow proceeds to block B305,where the image-alignment device obtains a reference image (e.g., animage of a defect-free object). Next, in block B310, the image-alignmentdevice preprocesses the image, thereby generating a preprocessedreference image 301. The preprocessing may include two operations. Thefirst operation removes (e.g., cuts out) invalid, unwanted, orirrelevant areas from the reference image. For example, thepreprocessing may remove the areas of the image in (A) in FIG. 4 thatare outside the bounding box 451. In FIG. 4, (B) shows the image afterthe areas that are outside the bounding box 451 have been removed. Thesecond operation generates a mask for the image. For example, the maskmay be generated based on an intensity threshold or on information aboutthe spatial structure of the object in the image. In FIG. 4, (C) showsan example of a mask, and (D) shows the image after the mask has beenapplied. In some embodiments, the reference image may be filtered (e.g.,a filter that sharpens edges).

After block B310, the first flow moves to block B315, which will bedescribed below.

From block B300, the second flow moves to block B330. In block B330, theimage-alignment device obtains an input image (e.g., a test image, whichis an image of an object that is being examined). Next, in block B335,the image-alignment device preprocesses the input image. Then the secondflow moves to block B340.

In block B340, the image-alignment device globally aligns the referenceimage and the input image. In some embodiments, the image-alignmentdevice uses subsampled images to obtain an approximate alignment. Forexample, in some embodiments, the preprocessed input image (e.g., asshown in FIG. 5(A) is subsampled, thereby generating a subsampled inputimage (e.g., as shown in FIG. 5(B). Also, the preprocessed referenceimage is subsampled, thereby producing a subsampled reference image. Theimage-alignment device then aligns the subsampled input image to thesubsampled reference image. To perform the alignment, some embodimentsof the image-alignment device use cross correlation. An example of thealignment result from cross-correlation is shown in FIG. 5(C), and aone-dimensional profile in the horizontal direction (x axis) of thecross correlation is shown in FIG. 5(D). The operations in block B340generate a global shift 302, which is a value that indicates a shift(e.g., in x and y values) that was performed on the preprocessed inputimage to align it with the preprocessed reference image, and aglobally-aligned input image, which is a preprocessed input image thathas been shifted by the global shift 302. Some embodiments use a morecomplex global alignment that may account for other imagetransformations beyond x and y shifts. For example, some embodimentsconsider in-plane rotation, image scaling, or image skew.

FIGS. 5(E) and 5(F) show the differences between a preprocessedreference image and a preprocessed input image before and after theglobal alignment.

The second flow then moves to block B345, where the image-alignmentdevice obtains subfields (e.g., patches, areas) in the globally-alignedinput image and the preprocessed reference image 301, for example bysplitting or dividing the images into equally-sized areas. In someembodiments, each subfield includes only a few pixels, and, in someembodiments, each subfield includes many pixels. Also, in someembodiments, the subfields overlap. FIG. 6 illustrates an exampleembodiment of an image that has been split into non-overlappingsubfields.

Next, in block B350, the image-alignment device performs multi-scalesubfield alignment (MSS alignment), which aligns each subfield in theglobally-aligned input image with the corresponding subfield in thereference image 301. Some embodiments of the image-alignment device usecross correlation to perform the alignment. Also, some embodiments ofthe image-alignment device use a weighted circular mask (e.g., as shownin FIG. 24, FIG. 26, or FIG. 27) to perform the cross correlation. FIG.7 illustrates an example of an image subfield that was aligned usingcross correlation. As shown in FIG. 7, a subfield of a processed inputimage (FIG. 7(A)) is compared with the

corresponding subfield of a reference image (FIG. 7(B)). Theimage-alignment device calculates the cross-correlation coefficients forall possible shifts, as shown in FIG. 7(C), and then identifies the peakof the cross-correlation coefficients. FIG. 7(C) shows the peak as across. FIG. 7(D) also shows the horizontal profile across the peak.Finally, FIGS. 7(E) and (F) show the intensity difference between thesubfield of the globally-aligned input image and the correspondingsubfield of the reference image before MSS alignment and after MSSalignment. FIG. 7(F) shows a significant improvement over FIG. 7(E). Theimage-alignment device performs the MSS alignment for multiplesubfields, and, in some embodiments, performs the MSS alignment for allthe subfields.

The operations of block B350 generate one or more MSS-alignment shifts303, each of which indicates the respective shift of a pixel or a groupof pixels (e.g., a subfield). FIG. 8 illustrates an example of theresults of an embodiment of MSS alignment. FIG. 8(A) shows the resultingMSS-alignment y shifts from selected subfields, and FIG. 8(B) shows theresulting MSS-alignment x shifts from selected subfields. In thisembodiment, the image-alignment device applied a smoothing process toidentify possible outliers and replaced the possible outliers based onnearby MSS-alignment shift values. The corresponding smoothedMSS-alignment shift results are shown in FIGS. 8(C) and (D). Also, someembodiments of the image-alignment device use interpolation to convertthe MSS-alignment x and y shifts of the subfields into respectiveMSS-alignment x and y shifts of the individual pixels, for example asshown in FIGS. 8(E) and (F).

The image-alignment device then shifts each pixel in theglobally-aligned input image by its corresponding MSS-alignment shift303, thereby producing an MSS-aligned input image. The MSS-aligned inputimage may be warped relative to the globally-aligned input image.

FIG. 9 illustrates an example of the results of an embodiment of MSSalignment. FIG. 9(A) illustrates the interpolated shifts, for selectedpixels, that were obtained from MSS alignment for the entire image. Thelines show the corresponding shifts between the input image and thereference image for each selected pixel. The differences between the twoimages before MSS alignment and after MSS alignment are respectivelyshown in FIGS. 9(B) and (C). Based on the magnitude of the reduction ofthe pixel intensity differences, the MSS alignment produced asignificant improvement. The MSS alignment can also be performed on aborder subfield, where only a portion of the subfield can be used tocalculate the cross correlation.

The second flow then proceeds to block B355.

From block B310, the first flow moves to block B315. In block B315, theimage-alignment device obtains (e.g., selects, identifies, generates)anchor points 304 (e.g., fixed-scale anchor points) in the preprocessedreference image. For example, the anchor points 304 may be selectedbased on the intersections of lines, the intersections of curves, or theintersections of high-order polynomials. As shown in FIG. 10 (whichshows the same image as FIG. 9(C)), although MSS alignment can producegood alignment results, in some circumstances MSS alignment will notcorrect all of the mismatches between two images. In FIG. 10, mismatchesare still visible in at least the areas within the rectangles that havesolid lines. Compared to the area within the rectangle that has dashedlines (the center rectangle), the other areas have noticeablemismatches. Also, simply repeating the MSS alignment with differentsizes of subfields (e.g., larger subfields, smaller subfields) may notreduce the mismatches. Thus, some embodiments of the image-alignmentdevice use anchor points 304 to reduce the mismatches.

FIG. 11 illustrates example embodiments of anchor points 304 in asubfield of an image. In this embodiment, the anchor points 304 are thepositions of intersections of lines, which may be identified from theedge points in the image. Unlike plane alignment, which has two axes offreedoms, and unlike line alignment, which has one axis of freedom, theintersections of lines have no freedom to shift on the x and y axes, andeach intersection generates a respective unique position, which can beused to shift images or image subfields into alignment.

FIG. 11(A) shows a subfield (e.g., a patch) from an image. FIG. 11(B)shows an edge image that was generated by performing edge detection onthe subfield that is shown in FIG. 11(A). The white stars in FIG. 11(B)are the anchor points 304, which were generated from the intersectionsof the lines. In some embodiments, the operations that obtain theseanchor points 304 split the subfield into many sub-subfields (e.g.,sub-patches) and then identify all the lines and curves in eachsub-subfield and the intersections of the lines and curves. Also, someembodiments move a sliding window over the image and identify all of theanchor points 304 that are currently visible in the sub-subfield that isframed by the sliding window. FIG. 11(C) shows examples of thesub-subfields and their respective intersections. For ease ofillustration, only a small number of the sub-subfields that containanchor points 304 are displayed in FIG. 11(C). FIG. 12 shows an exampleof the anchor points 304 in a complete image.

After the image-alignment device has obtained the anchor points 304 inblock B315, the first flow moves to block B320. In block B320, theimage-alignment device obtains anchor-edge points 305, which aredynamic-scale anchor-edge points 305 in this example embodiment.Depending on the spatial structure of the object in the image, therecould be some areas in the image that contain few or no anchor points304. In such areas, the x and y shifts (in pixels) may need to beinterpolated based on the anchor points 304. This interpolation mayreduce the precision of the alignment.

Thus, some embodiments use anchor-edge points 305 as second-levelpivots, which may eliminate or reduce such interpolation. Anchor-edgepoints 305 are edge points (e.g., points on a line segment that are noton a corner) that were detected by the edge detection. Instead of usingthe intensively-sampled edge points as anchor-edge points 305, someembodiments use just a few (e.g., one, two) edge points in one smallsub-subfield (e.g., 8 by 8 pixels). Each anchor-edge point 305 may beassociated with one or more anchor points 304 that were obtained inblock B315, and each anchor point 304 may be associated with one or moreanchor-edge points 305. FIG. 13 illustrates example embodiments ofanchor-edge points 305. FIG. 13(A) shows the anchor points 304 (inlighter gray) and the anchor-edge points 305 (in darker gray).

The first flow then moves to block B325, where the image-alignmentdevice obtains neighborhood areas 306 for the anchor-edge points 305. Insome embodiments, a neighborhood area 306 contains at least two to fouranchor points 304. A neighborhood area 306 with more than two anchorpoints 304 may guarantee that there is enough spatial information foralignment. Some of the anchor-edge points 305 and their associatedneighborhood areas 306 are shown in FIG. 13(B). The neighborhood areas306 do not need to be identical in some embodiments—thus, they can havedifferent shapes, different sizes, etc.

Then the first flow ends in block B360.

Once the second flow has moved to block B355, the image-alignment deviceuses the anchor points 304, the anchor-edge points 305, and theneighborhood areas 306 to perform anchor-point multi-scale-subfieldalignment (APMSS) on the preprocessed reference image 301 and on theMSS-aligned input image. The image-alignment device aligns each of theneighborhood areas 306 in the MSS-aligned input image with thecorresponding neighborhood area in the preprocessed reference image 301.To perform the alignment of the neighborhood areas 306, theimage-alignment device may use cross correlation on the anchor points304 and on the anchor-edge points 305. Some embodiments of theimage-alignment device use a weighted circular mask (e.g., as shown inFIG. 24, FIG. 26, or FIG. 27) to perform the cross correlation.

The operations of block B355 generate one or more APMSS-alignment shifts307, each of which indicates the respective shift of a pixel or a groupof pixels (e.g., a subfield). The image-alignment device then shiftseach pixel in the MSS-aligned input image by its correspondingAPMSS-alignment shift 307, thereby producing an APMSS-aligned inputimage 308. The APMSS-aligned input image may be warped relative to theglobally-aligned input image and the MSS-aligned input image.

Then the image-alignment device may send the APMSS-aligned input imageto another device, store the APMSS-aligned input image, or perform otheroperations on or with the APMSS-aligned input image (e.g., use thealigned images of non-defective images to build nominal object models,search for defects in the APMSS-aligned input image, or cause theAPMSS-aligned image to be displayed on a display device). The flow thenends in block B360.

FIG. 14 illustrates example embodiments of the differences between areference image and an MSS-aligned image and the differences between areference image and an APMSS-aligned image. The differences between thereference image and the MSS-aligned image are shown on the left, and thedifferences between the reference image and the APMSS-aligned image areshown on the right. The APMSS-aligned image shows an improvementrelative to the MSS-aligned image.

FIG. 15A illustrates an example embodiment of pixel shifts. FIG. 15Ashows the starting positions of three pixels, which are then shiftedaccording to one or more pixel shifts (e.g., a global-alignment shift,an MSS-alignment shift, an APMSS-alignment shift). FIG. 15A also showsthe ending positions of the three pixels. As shown by FIG. 15A, each ofthe pixels may be shifted differently, so that the distances between thepixels and the spatial orientations of the pixels changes. These changesmay warp the appearance of the image.

FIG. 15B illustrates an example embodiment of pixel interpolation. Aftertwo pixels are shifted from their starting points (SP) to their endingpoints (EP), the distance between the pixels changes. In this example,the two pixels abutted each other in their starting points, but areseparated by three pixels once they are shifted to their ending points.Also, in some circumstances, other pixels may not be shifted into thelocations of these three pixels. Accordingly, after shifting pixels, theimage-alignment device may interpolate the pixels in the post-shiftimage for which the image-alignment device does not have data.

FIG. 16 illustrates an example embodiment of a mapping of a pixel froman APMSS-aligned input image to an unaligned input image. To identifythe pixel in an unaligned image that corresponds to a pixel in a shiftedimage (e.g., a globally-aligned image, an MSS-aligned image, andAPMSS-aligned image), the image-alignment device can use the shifts thatwere generated by the alignment operations to map the pixel in theshifted image to the corresponding pixel in the unaligned image (e.g.,an input image). This example shows a pixel in an APMSS-aligned imagethat was shifted by global alignment, MSS alignment, and APMSSalignment. The position in the APMSS-aligned image is the aggregate ofthe three shifts. Using the global shift 1602, the MSS-alignment shift1603, and the APMSS-alignment shift 1607, the image-alignment device canmap the pixel from its position in the APMSS-aligned image to itsposition in the unaligned image.

FIG. 17 illustrates an example embodiment of an operational flow foraligning images. The operational flow starts in block B1700 and thensplits into a first flow and a second flow. The first flow proceeds toblock B1705, where the image-alignment device obtains a reference image(e.g., an image of a defect-free object). Next, in block B1710, theimage-alignment device preprocesses the image, thereby generating apreprocessed reference image 1701. After block B1710, the first flowmoves to block B1715, which is described below.

From block B1700, the second flow moves to block B1730. In block B1730,the image-alignment device obtains an input image (e.g., a test image,which is an image of an object that is being examined). Next, in blockB1735, the image-alignment device preprocesses the input image. Then thesecond flow moves to block B1740.

In block B1740, the image-alignment device globally aligns the referenceimage and the input image. The operations in block B1740 generate aglobal shift 1702 and a globally-aligned input image.

The second flow then moves to block B1745, where the image-alignmentdevice obtains subfields (e.g., patches) in the globally-aligned inputimage and the preprocessed reference image 1701. Next, in block B1750,the image-alignment device performs multi-scale subfield alignment (MSSalignment), which aligns each subfield in the globally-aligned inputimage with the corresponding subfield in the preprocessed referenceimage 1701. The operations of block B1750 generate one or moreMSS-alignment shifts 1703. The image-alignment device then shifts eachpixel in the globally-aligned input image by its correspondingMSS-alignment shift 1703, thereby producing an MSS-aligned input image.The second flow then proceeds to block B1755.

From block B1710, the first flow moves to block B1715. In block B1715,the image-alignment device obtains (e.g., selects, identifies,generates, detects) anchor points 1704 (e.g., static anchor points,dynamic anchor points) and anchor-point neighborhood areas 1706 in thepreprocessed reference image 1701. Some embodiments of theimage-alignment device obtain anchor points that are (1) easilyidentifiable, and (2) good alignment points. Some embodiments use cornerpoints or line intersections where the lines create corners that have asignificant angle (e.g., are not parallel or are not nearly parallel).In some circumstances, it is easy to align two or more image linestructures in the directions that are orthogonal to the line structures,but not easy to align the image line structures in the directions thatare parallel to the line structures. However, if a subfield includes twonon-parallel lines, the non-parallel lines will provide strong alignmentresults in two non-parallel directions, and thus the alignment objectivefunction (e.g., based on cross-correlation) will be well posed. If thelines were parallel, the objective function would have one degree offreedom (for example, be alignable only in the x direction, not in the ydirection), and the alignment problem would be ill-conditioned.

When obtaining the anchor points, some embodiments of theimage-alignment device validate the detected anchor points based on aninput image. In some of these embodiments, the anchor-point location isshifted by small amounts and cross correlations are performed betweenthe input image and the reference image in the neighborhood area of theshifted anchor point. If the anchor point is good (e.g., stable inalignment), the local alignments found from the shifted anchor pointsshould be consistent with each other after compensating for the shiftamounts. For example, if the image-alignment device shifts the anchorpoint in the x direction by 1 pixel and performs an alignment of theshifted anchor point in the reference image to the input image, therequired shift should be the same as the alignment shift found from theoriginal anchor point. Also, these embodiments may examine theconsistency and stability of the alignment in the local neighborhoodarea. Some embodiments examine the standard deviation of the detectedshifts (compensated for the anchor-point shift) across several smallshifts of the anchor point. If the standard deviation is small, theanchor point is deemed to be stable and good. On the other hand, if thestandard deviation is large, the anchor point is discarded due to itsinconsistent alignment results. Some example embodiments use four shiftson the anchor points (shift in x by +/− 1 and shift in y by +/− 1).Other example embodiments use eight shifts: (−1,−-1), (−1,0), (−1,1),(0,−1), (0,1), (1,−1), (1,0), and (1,1).

And, in block B1715, some embodiments of the image-alignment device findthe points in an image that are easily self-alignable. A point that iseasily self-aligned using autocorrelation may be a good point to alignvia cross-correlation with another image if the images are somewhatsimilar.

In some embodiments, the autocorrelation of a two-dimensional (2D)neighborhood area of size 2 w×2 w centered at (x₀, y₀) for a function f(x, y) may be described by

h _((x) _(0,) _(y) ₀ ₎ (x, y)=∫_(−w) ^(w)∫_(−w) ^(w) f(u−x ₀ , v−y₀)f(u−x ₀ +x, v−y ₀ +y) du dv.   (1)

In the case of digital images, the discrete analogue for a (2 w+1)×(2w+1) window size may be described by

$\begin{matrix}{{h_{({x_{0},y_{0}})}( {x,y} )} = {\sum\limits_{v = {- w}}^{w}{\sum\limits_{u = {- w}}^{w}{{f( {{u - x_{0}},{v - y_{0}}} )}{{f( {{u - x_{0} + x},{v - y_{0} + y}} )}.}}}}} & (2)\end{matrix}$

In some embodiments, the auto-correlation is essentially the dot productof the windowed function (pixel values) with itself over a variety ofshifts.

Some embodiments of the autocorrelation can be computed using a FastFourier Transform. In these embodiments, the shifting of the second termin equation (2) is performed cyclically. Thus, two subfields undergoingthe dot product have the same magnitude. According to theCauchy-Schwartz Inequality, this implies that the autocorrelationfunction will be maximum at the zero shift and will be less than orequal to the maximum everywhere else. This further implies that theautocorrelation function should be (non-strictly) concave around thezero shifts (An auto-correlation function computed with cyclic shiftsmay be concave or flat.).

For example, FIG. 18 illustrates an example embodiment of the discreteauto-correlation for two different points in an image. The left graph inFIG. 18 shows a point that is not conducive to alignment because thepoint presents itself as a ridge-like structure in one direction of theautocorrelation function of a neighborhood area around that point. Theridge indicates that if this point was to be aligned to itself, it isharder to align in one direction (the ridge direction). However, theright graph shows a strong peak structure, which indicates that it wouldbe easy to align this point to itself.

One way to characterize these observations uses the peakiness of theautocorrelation curve. Some embodiments calculate the Hessian matrix ofthe autocorrelation at the center pixel. As a reminder, the Hessian of atwo-dimensional function is the operator on that function:

$\begin{matrix}{H = {\begin{bmatrix}\frac{\partial^{2}}{\partial x^{2}} & \frac{\partial^{2}}{{\partial x}{\partial y}} \\\frac{\partial^{2}}{{\partial x}{\partial y}} & \frac{\partial^{2}}{\partial y^{2}}\end{bmatrix}.}} & (3)\end{matrix}$

In particular, some embodiments take advantage of the Hessian at thecenter of the autocorrelation function.

In a continuous autocorrelation function, the Hessian will be anon-positive-definite matrix. This indicates that the eigenvalues of thematrix should be less than or equal to zero. These eigenvalues indicatethe concavity of the autocorrelation curve in the most concave directionand the least concave direction. In the left example in FIG. 18, theweakest concavity occurs in the direction of the ridge, and someembodiments use this value to represent the worst-case alignability ofthat point. Thus, because the autocorrelation function is always concaveor flat, the largest eigenvalue of the Hessian should be negative orzero. By negating this eigenvalue, some embodiments generate a score forthat point that defines the point's relative alignability.

Some embodiments then generate an alignability map based on the largesteigenvalue of the Hessian of the autocorrelation matrix for a respectiveneighborhood area around each point in the image.

Because using “brute-force” techniques to perform this calculation canbe computationally expensive, some embodiments use other techniques.First, note that estimating the second order partial derivatives of theauto-correlation at the zero alignment point does not require all pointsof the auto-correlation to be known.

The second derivative can be estimated by a second order difference.Consider the sequence {h₁, h₂, h₃, . . . , h_(n)}. The first differenceof this sequence provides a new sequence: {h₂−h₁, h₃−h₂, h₄'h₃, . . . ,h_(n+1)−j_(n)}. The second difference of this sequence provides a newsequence:

{h ₃−2h ₂ +h ₁ , h ₄−2h ₃ +h ₂ , . . . , h _(n+1)−2h _(n−1)}.   (4)

Thus, the second order difference at the center depends only on thecenter value of the auto-correlation and the two immediate neighbors inthe direction of the difference. So the second order differences in thex and y directions can be calculated from these values:

$\begin{matrix}\begin{matrix}\; & h_{0,{- 1}} & \; \\h_{{- 1},0} & h_{0,0} & {h_{1,0}.} \\\; & h_{0,1} & \;\end{matrix} & (5)\end{matrix}$

Additionally, there are redundant calculations when computingauto-correlation functions centered at different pixels. Theautocorrelations are dot products of pixel values in a specified window.The dot product is the sum of the element-by-element products in thewindow. The element-by-element products may be pre-computed for thefive-shift configurations shown in equation (5) across the whole image.The element-by-element products for each shift (s, t) from equation (5)over the entire image may be described by

p _(s,t)(x, y)=f(x, y)f(x−s, y−t).   (6)

Moreover, some embodiments use integral images to determine the dotproducts over the windows of interest. Some embodiments of an integralimage can be described by the following:

$\begin{matrix}{{I_{p_{s,t}}( {x,y} )} = {{\sum\limits_{u = 1}^{x}{\sum\limits_{v = 1}^{y}{p_{s,t}( {u,v} )}}} = {{I_{p_{s,t}}( {x,{y - 1}} )} + {{p_{s,t}( {x,y} )}.}}}} & (7)\end{matrix}$

The integral image can be computed in a running accumulation with asingle pass over the image p, as can be seen in the last term ofequation (7).

In some embodiments, to then find a windowed dot product, thecomputation of the windowed autocorrelation can be described by thefollowing:

h _(s,t,w)(x, y)=I _(p) _(s,t) (x+w, y+w)−I _(p) _(s,t) (x−w−1, y+w)−I_(p)(x+w, y−w−1)+I _(p) _(s,t) (x−w−1, y−w−1).   (8)

Furthermore, in some embodiments, non-cyclic auto-correlationcalculations are normalized to calculate dot products of unit vectors.Due to the shifts, the shifted windows (e.g., patches, areas, subfields)may vary in magnitude from window to window. Thus, some embodimentsnormalize the window by the magnitude of the window to create unitvectors. Concerning the magnitude of each window in the image, note thatthe magnitude squared of each window is the sum of the squares of theimage intensities in the window.

This can be computed from the integral image of the zero shifts. Thus,in some embodiments, the normalized auto-correlations, denoted it can bedescribed by the following:

$\begin{matrix}{\; {\begin{matrix}\; & \frac{h_{0,{- 1},w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {x,{y - 1}} )}}} & \; \\\frac{h_{{- 1},0,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x - 1},y} )}}} & 1 & {\mspace{14mu} \frac{h_{1,0,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x + 1},y} )}}}} \\\; & \frac{h_{0,1,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {x,{y + 1}} )}}} & \;\end{matrix}.}} & (9)\end{matrix}$

The normalization guarantees that the term in the center is one and theterms above, below, left, and right are all less than or equal to one.

Some embodiments approximate the Hessian with a diagonal matrix. Thecenter point of the auto-correlation is a local maxima of theauto-correlation function. Because the above normalization guaranteesthat the center point of the auto-correlation is a local maxima or atleast equal to its neighbors, the first derivative at the zero-shiftposition is zero. Consequently, the off-diagonal terms of the Hessian ofthe auto-correlation at the zero shift position are zero:

$\begin{matrix}{{{\frac{\partial^{2}}{{\partial x}{\partial y}}h} = 0}.} & (10)\end{matrix}$

This is a necessary condition for a local maximum.

Additionally, the eigenvalues of a diagonal matrix are the diagonalelements. Thus the Hessian eigenvalues are the second differences of theauto-correlation function in both x and y:

$\begin{matrix}{{{{\lambda_{1}( {x,y} )} = {\frac{h_{{- 1},0,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x - 1},y} )}}} + \frac{h_{1,0,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x + 1},y} )}}} - 2}};}\mspace{20mu} {and}} & (11) \\{{\lambda_{2}( {x,y} )} = {\frac{h_{0,{- 1},w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {x,{y - 1}} )}}} + \frac{h_{0,1,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {x,{y + 1}} )}}} - 2.}} & (12)\end{matrix}$

Both eigenvalues are non-positive. The worst-case concavity direction isthe direction of the larger eigenvalue (the one closer to zero). Thus,some embodiments create a worse-case concavity score E, based on thenegative of the largest eigenvalue, for example as described by thefollowing:

E(x, y)=−max {λ₁(x, y), λ₂(x, y)}.   (13)

However, some of the previous embodiments may not be ideal when theautocorrelation forms a diagonal ridge-like structure—in which case theapproximation may break down. Thus, some embodiments solve for theHessian by using the nine points around the center of theautocorrelation and fitting them to a two-dimensional second orderpolynomial. In some embodiments, this may be described by the followingfunction:

g(x, y)=a+bx+cy+dxy+ex ² +f y ².   (14)

This may be described by the following matrix M for x from −1 to 1 and yfrom −1 to 1.

1 x y xy x² y² 1 −1 −1 1 1 1 1 −1 0 0 1 0 1 −1 1 −1 1 1 1 0 −1 0 0 1 1 00 0 0 0 1 0 1 0 0 1 1 1 −1 −1 1 1 1 1 0 0 1 0 1 1 1 1 1 1

Then, some embodiments fit the existing normalized autocorrelations asdescribed by the following:

{tilde over (h)}≈M·[a b c d e f ] ^(T).   (15)

Also, the Hessian matrix for g(x,y) in Equation (14) may be described bythe following:

$\begin{matrix}{H_{g} = {\begin{bmatrix}{2e} & d \\d & {2f}\end{bmatrix}.}} & (16)\end{matrix}$

The coefficients d, e, and f can be found using a least-squares fit fromthe last three rows of the pseudoinverse of M, for example as describedby the following:

[a b c d e f] ^(T)=(MM ^(T))⁻¹ M ^(T) {tilde over (h)}.   (17)

Because M is known and given in the table above, in some embodiments theoptimal weightings of the normalized autocorrelation for the d term in(16) can be described by the following:

$\begin{matrix}{{\frac{1}{4}\begin{bmatrix}1 & 0 & {- 1} \\0 & 0 & 0 \\{- 1} & 0 & 1\end{bmatrix}}.} & (18)\end{matrix}$

Thus,

$\begin{matrix}{{\frac{\partial^{2}}{{\partial x}{\partial y}}\overset{˜}{h}} = {\frac{h_{{- 1},{- 1},w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x - 1},{y - 1}} )}}} + \frac{h_{1,{1w}}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x + 1},{y + 1}} )}}} - \frac{h_{{- 1},1,w}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x - 1},{y + 1}} )}}} - {\frac{h_{1,{- 1}}( {x,y} )}{\sqrt{{h_{0,0,w}( {x,y} )} \cdot {h_{0,0,w}( {{x + 1},{y - 1}} )}}}.}}} & (19)\end{matrix}$

Some embodiments of the weightings for the 2e term of equation (16) maybe described by the following:

$\begin{matrix}{{\frac{1}{3}\begin{bmatrix}1 & {- 2} & 1 \\1 & {- 2} & 1 \\1 & {- 2} & 1\end{bmatrix}}.} & (20)\end{matrix}$

And some embodiments of the weightings for the 2f term of equation (16)may be described by the following:

$\begin{matrix}{{\frac{1}{3}\begin{bmatrix}1 & 1 & 1 \\{- 2} & {- 2} & {- 2} \\1 & 1 & 1\end{bmatrix}}.} & (21)\end{matrix}$

Note that there are only six unknowns in the parametric polynomialmodel, and some embodiments use nine points around the center of theautocorrelation function. Thus, some embodiments reduce the number ofpoints used to as low as six to reduce the computational expense.

The largest eigenvalue can be found from the characteristic equation ofthe Hessian matrix, which leads to the following:

$\begin{matrix}{\lambda_{\max} = {\frac{( {{\frac{\partial^{2}}{\partial x^{2}}\overset{˜}{h}} + {\frac{\partial^{2}}{\partial y^{2}}\overset{˜}{h}}} ) + \sqrt{( {{\frac{\partial^{2}}{\partial x^{2}}\overset{˜}{h}} - {\frac{\partial^{2}}{\partial y^{2}}\overset{˜}{h}}} )^{2} + {4( {\frac{\partial^{2}}{{\partial x}{\partial y}}\overset{˜}{h}} )^{2}}}}{2}.}} & (22)\end{matrix}$

The resulting alignability map E may be described by the following:

E(x, y)=−λ_(max).   (23)

One of the previous embodiments was applied to an x-ray image of awheel. The results from one embodiment are illustrated in FIG. 19. InFIG. 19, a wheel x-ray image is shown on the right, and thecorresponding alignment point-map is shown on the left. Some peaks inthe alignment-point map are marked with asterisks, and theircorresponding locations are shown on the x-ray image.

Accordingly, some embodiments generate the five to nine images that aredescribed by equations (5), (18), (20), and (21). Each of these imagesmay be generated (e.g., computed) in a number of operations proportionalto the number of pixels in the original image. From the five to nineintermediate images, two eigenvalues may be calculated, and the largerof the two eigenvalues at each pixel may then be negated and used togenerate an alignability map (a map of the relative alignability of eachpoint in the image).

Also, further processing of the alignability maps E of equations (13)and (23) may be done to detect peaks in the map.

Moreover, these embodiments are not limited to the presence of straightlines. Curved lines that span multiple dimensions may reveal themselvesto be alignable features in an image, depending on the window size.

The use of integral images may enable these operations to be easily andefficiently repeated on multiple scales. Equation (8) shows that theanalysis scale w can be readily changed to quickly produce the five tonine images of equations (5), (18), (20), and (21).

In some embodiments, a plurality of alignability maps E are generatedacross a plurality of scales, and the detection of alignment points isdetermined based on the plurality of alignability maps E across theplurality of scales.

Another benefit of these embodiments is that the stability of thealignment points can be ascertained based on a single image example.These embodiments do not need to use cross-correlations with otherimages, as the original image contains appropriate information regardingthe alignable structures in the image.

Also, given an alignability map (e.g., a map generated from the largesteigenvalue of the Hessian of the autocorrelation matrix) that has afixed window size for autocorrelation, some embodiments of theimage-alignment device select anchor points by using all local maximasin the alignability map as anchor points. However, anchor-pointselection based only on local maxima may result in a dense map and maylimit the control over the uniformity in the distribution of the anchorpoints. In contrast, in some embodiments, an ideal distribution foranchor points is an almost uniform distribution of anchor points, whichmay improve the accuracy of alignment across the whole image.Local-maxima-based anchor-point selection may result in a highcomputational load for alignment, due to the large number of anchorpoints detected. Thus, to give control over the distribution of theanchor points, some embodiments divide the alignability map into windowsof equal size, and select at least one anchor point within each window,if available.

Some embodiments of the image-alignment device select the location ineach window of the alignability map where the alignability value is amaximum. These values are often good candidates for anchor points. Also,some embodiments also have a minimum threshold for the values in thealignability map, and the threshold may be set based on the noise levelin the image. This threshold may help to avoid selecting anchor pointsthat are artefacts of the noise in the image. However, this can cause anumber of windows to remain empty if no anchor points in the windowswere selected due to low values in the alignability map in thosewindows.

Additionally, the alignment-map profile may depend on the size of theautocorrelation window used in the computing of the autocorrelation. Theautocorrelation equation for a discrete image for a (2 w+1)×(2 w+1)window size can be described by the following:

$\begin{matrix}{{h_{({x_{0},y_{0},w})}( {x,y} )} = {\sum\limits_{v = {- w}}^{w}{\sum\limits_{u = {- w}}^{w}{{f( {{u - x_{0}},{v - y_{0}}} )}{{f( {{u - x_{0} + x},{v - y_{0} + y}} )}.}}}}} & (24)\end{matrix}$

Equation (24) shows that the autocorrelation is a function of windowsize (2 w+1)×(2 w+1). The scale of an anchor point may be the windowsize of the auto correlation. Good features that are smaller than thewindow size may be well highlighted in the alignability map.Accordingly, regions in the image where the alignability map is weak inone window size may get a stronger alignability map in a differentwindow size, for example as illustrated in FIG. 20, which illustratesexample embodiments of alignability maps for different window sizes.

Consequently, the information from the alignability maps for differentwindow sizes can be combined to get a set of anchor points with scalesselected for cross correlation with a test image. Some embodiments usethe same autocorrelation-window size that was used for autocorrelationin the reference image when performing cross correlation in the inputimage (the image to be aligned to the reference image). An anchor pointmay be defined by the location and scale (size of the window) for crosscorrelation. An example embodiment of an operational flow to obtainanchor points from alignability maps is illustrated in FIG. 21.

After block B1715, the first flow ends in block B1760.

Once the second flow has moved to block B1755, the image-alignmentdevice uses the anchor points 1704 and the neighborhood areas 1706(e.g., autocorrelation window) to perform anchor-pointmulti-scale-subfield alignment (APMSS) on the preprocessed referenceimage 1701 and on the MSS-aligned input image. The image-alignmentdevice aligns each of the neighborhood areas 1706 in the MSS-alignedinput image with the corresponding neighborhood area in the preprocessedreference image 1701

The operations of block B1755 generate one or more APMSS-alignmentshifts 1707, each of which indicates the respective shift of a pixel ora group of pixels (e.g., a subfield, a sub-subfield). Theimage-alignment device then shifts each pixel in the MSS-aligned inputimage by its corresponding APMSS-alignment shift 1707, thereby producingan APMSS-aligned input image 1708. The APMSS-aligned input image 1708may be warped relative to the globally-aligned input image and theMSS-aligned input image.

Then the image-alignment device may send the APMSS-aligned input image1708 to another device, store the APMSS-aligned input image 1708, orperform other operations on or with the APMSS-aligned input image 1708(e.g., search for defects in the APMSS-aligned input image 1708, causethe APMSS-aligned input image 1708 to be displayed on a display device).The flow then ends in block B 1760.

Additionally, applications of the anchor-point operations extend beyondimage alignment. As a general key-point detection method, theseoperations have many useful properties. First, the operations are fastand of computation order O(N), where N is the number of pixels. Second,the operations seek out points that are self-identifiable (i.e., noteasily confused with neighboring points). The identification ofkey-points can be used for general object recognition, image alignmentfor image stitching, medical imaging, non-destructive testing, etc.Additionally, the key-points may be used for image fingerprinting, imagematching, and image search.

FIG. 21 illustrates an example embodiment of an operational flow toobtain anchor points from alignability maps. In block B 2100, theimage-alignment device obtains M alignability maps (map₁, map₂,map_(M)), which are arranged in order of scale such that map₁ has thesmallest scale and map_(M) has the largest scale. The respective scaleof an alignability map indicates the size of the autocorrelation windowthat was used to generate the alignability map. Next, in block B 2105,the image-alignment device divides all the alignment maps into windows,for example using a grid, and sets a window counter n to 1 (n=1). Thewindows may be the same size and have corresponding positions. Forexample, some embodiments divide each alignability map into N>>1equally-sized windows, and window n=1 has the same position in thetop-left corner in all of the alignability maps.

The flow then moves to block B2110, where the image-alignment deviceobtains an upper threshold and a lower threshold. The flow then proceedsto block B2115, where the image-alignment device sets the map counter mto 1 (m=1). Then, in block B2120, the image-alignment searchesalignability map m for the maximum value in the window n. In blockB2125, the image-alignment device determines if the maximum alignabilityvalue (MAV) found in block B2120 for the window n exceeds the upperthreshold. If the MAV exceeds the upper threshold (block B2125=Yes),then the flow moves to block B2130. In block B2130, the image-alignmentdevice saves the location of the maximum alignability value in window nas an anchor point with scale m (the scale indicates or defines theneighborhood area of the anchor point). Then the flow proceeds to blockB2175.

If an anchor point was not found (block B2125=No), then the flow movesto block B2135. In block B2135, the image-alignment device determines ifit has searched all of the alignability maps. If it has not (blockB2135=No), then the flow moves to block B2140, where the image-alignmentdevice sets the map counter m to the next map (m=m+1), and then the flowreturns to block B2120. If the image-alignment device has searched allof the alignability maps (block B2135=Yes), then the flow proceeds toblock B2145.

In block B2145, the image-alignment device sets the map counter m to 1(m=1). Next, in block B2150, the image-alignment device searchesalignability map m for the maximum value in the window n. In blockB2155, the image-alignment device determines whether the MAV found inblock B2150 for the window n exceeds the lower threshold. If the MAVexceeds the lower threshold, (block B2155=Yes), then the flow moves toblock B2160. In block B2160, the image-alignment device saves thelocation of the maximum alignability value in window n as an anchorpoint with scale m, and the image-alignment device may define the anchorpoint by the location of the maximum alignability value and the scale ofthe alignability map m. The scale indicates or defines the neighborhoodarea of the anchor point. Then the flow proceeds to block B2175.

If an anchor point was not found (block B2155=No), then the flow movesto block B2165. In block B2165, the image-alignment device determines ifit has searched all of the alignability maps. If it has not search allof the alignability maps (block B2165=No), then the flow moves to blockB2170, where the image-alignment device sets the map counter m to thenext map (m=m+1), and then the flow returns to block B2150. If theimage-alignment device has searched all of the alignability maps (blockB2165=Yes), then the flow proceeds to block B2175.

In block B2175, the image-alignment device determines if it has searchedall of the windows. If the image-alignment device determines that it hasnot searched all of the windows (block B2175=No), then the flow moves toblock B2180, where the image alignment devices sets the window counter nto the next window (n=n+1), and then the flow returns to block B2115.Otherwise (block B2175=Yes), the flow ends in block B2185.

Thus, for a window, if no location exists in an alignment map where thealignment value exceeds the upper threshold for all the alignability-mapscales, the image-alignment device searches for the smallestalignability-map scale that has a maximum value that exceeds the lowerthreshold. If such a point in the window exists, the anchor point isdefined by the location of the maximum value and the smallestalignability-map scale that has the point that exceeds the lowerthreshold value.

And if the image-alignment device cannot find a location in the windowwhere the alignment value exceeds the lower threshold, theimage-alignment device does not select any anchor point within thatwindow.

Thus, some embodiments, like the embodiment in FIG. 21, use twothresholds (upper and lower) to give preference to the smallest-scalefeature that exceeds a minimum quality. These operational flowsinitially search for a higher threshold, indicating a higher-qualityanchor point, from the smallest alignability-map scale. For the windowsin the alignability-map scale that do not have any point that exceedsthe upper threshold, the image-alignment device searches for the anchorpoint using a lower threshold value that is set above the noise level ofthe alignability map. If none of the alignability maps for a window havea point that exceeds the lower threshold, the image-alignment devicestops without finding any anchor point for that window. Additionally,when some embodiments do not find an alignability value above the higherthreshold, they use the scale of alignability with the highestalignability value found, provided it is above the lower threshold.

FIG. 22 illustrates example embodiments of anchor points. The differentshapes for the anchor points indicate the different window sizes (whichmay be the same as the sizes of the anchor points' correspondingneighborhood areas) associated with those anchor points.

FIG. 23 illustrates example embodiments of anchor points and anchor-edgepoints in a one-dimension signal. The signal is unevenly distributedacross the horizontal axis. To fully characterize the features of thesignal for alignment, some embodiments use the turning points of thecurve, plotted as the black stars, as anchor points. However, using onlythe anchor points to interpolate all the others may produce large errorsin the regions between the anchor points. Thus, some embodiments useanchor-edge points, shown as black dots, which serve as a second layerof anchoring positions during image alignment. The combination of anchorpoints and anchor-edge points permits some embodiment to handle bothslowly-changing areas (for example, the area of ‘a’) and fast-changingareas (the area of ‘b’). The corresponding regions for anchor-edgepoints could be varied depending on the neighboring anchor points.

FIG. 24 illustrates an example embodiment of a weighted circular mask2471 that may be used for cross correlation. In FIG. 24, S_(s) is thelength of the subfield (and the width if the subfield is a square), R isthe inner-circle radius of the mask 2471, and M is the outer-circleradius of the mask. The pixels in the inner-circle radius R contributefully to the cross correlations, the pixels in the annulus between theinner-circle radius R and the outer-circle radius M contribute onlypartially to the cross correlations, and the pixels that are outside theouter-circle radius M do not contribute to the cross correlations. Inthis embodiment, the inner-circle radius R is equal to half the lengthS_(s) of the subfield. Also, the contribution of a pixel in the annulusmay decrease linearly with distance from the inner-circle radius R.

The weighted circular mask 2471 may help to compensate for the rotationwhen two subfields are rotated relative to each other. FIG. 25Aillustrates example embodiments of a reference-image subfield and aninput-image subfield. The input-image subfield is rotated relative tothe reference-image subfield. FIG. 25B highlights the circular centralareas of the reference-image subfield and the input-image subfield. FIG.25C highlights the boundary (edge) areas of the input-image subfield. Asshow in FIGS. 25B and 25C, the differences between the reference-imagesubfield and the input-image subfield are larger at the boundary areas.

The rotations encountered during MSS alignment are often small, and evena pixel-by-pixel comparison of the subfields may be adequately reliable.However, the reliability of the pixel-to-pixel comparison will decreasefor the pixels that are farther from the center of the subfield.Specifically, a pixel's location will vary by a distance of

${2r\mspace{14mu} \sin \frac{\theta}{2}},$

which is approximately rθ when θ is small (where r is the distance fromthe center of the subfield and θ is the angle of rotation in radians).

Consequently, some embodiments weight the cross-correlations of thesubfields so that pixels close to the center of the subfield areweighted as more important or reliable, while pixels further from thecenter are weighted less.

For example, in some embodiments, the non-weighted cross-correlationused to align a subfield in image f with a subfield in image g may bedescribed by

$\begin{matrix}{{{h_{({x_{0},y_{0}})}( {x,y} )} = {\sum\limits_{v = {- w}}^{w}{\sum\limits_{u = {- w}}^{w}{{f( {{u - x_{0}},{v - y_{0}}} )}{( {{u - x_{0} + x},{v - y_{0} + y}} )}}}}},} & (25)\end{matrix}$

where (x₀, y₀) is the subfield center in image f, and where (x, y) is ashift applied to image g. Also some cross-correlations normalize theimage subfields.

In some embodiments, the cross correlation applies a weighting functionw(u, v), for example as described by the following:

$\begin{matrix}{{h_{({x_{0},y_{0}})}( {x,y} )} = {\sum\limits_{v = {- w}}^{w}{\sum\limits_{u = {- w}}^{w}{{\omega ( {u,v} )}{f( {{u - x_{0}},{v - y_{0}}} )}{{( {{u - x_{0} + x},{v - y_{0} + y}} )}.}}}}} & (26)\end{matrix}$

Also, some embodiments of the weighting function w(u, v) can bedescribed by the following:

$\begin{matrix}{{W( {x, y \middle| x_{c} ,y_{c}} )} = \{ {{{\begin{matrix}{1,} & {r \leq R} \\{\frac{M - r}{M - R},} & {{R < r < M},} \\{0,} & {r \geq M}\end{matrix}{where}r} = \sqrt{( {x - x_{c}} )^{2} + ( {y - y_{c}} )^{2}}},} } & (27)\end{matrix}$

and where the circular weighted mask is centered at x_(c), _(c).

Some embodiments use other weighting functions. For example, someembodiments of the weighting function can be described by one of thefollowing:

$\begin{matrix}{{{W( {x, y \middle| x_{c} ,y_{c}} )} = {\exp( {{{- a}\; r},}\  )}},} & (28) \\{{{W( {x, y \middle| x_{c} ,y_{c}} )} = {\exp ( {{- a}r^{2}} )}},{and}} & (29) \\{{W( {x, y \middle| x_{c} ,y_{c}} )} = \{ {\begin{matrix}{{M - r},} & {r < M} \\{0,} & {r \geq M}\end{matrix}.} } & (30)\end{matrix}$

In some embodiments, the mask center point x_(c), y_(c) is a real valuerepresenting a center that is not necessarily at an exact pixellocation. In this way, the mask may be centered so that contributionsfrom the region around the center reflect fractional pixel shifts to thecenter.

An example of the weight values of each pixel of an embodiment of acircular weighted mask are shown in Table 1. The subfield size S_(s) is11 pixels, the inner-circle radius R is 4 pixels, and the outer-circleradius M is 6 pixels.

TABLE 1 0 0 0.084524 0.307418 0.45049 0.5 0.45049 0.307418 0.084524 0 00 0.171573 0.5 0.763932 0.938447 1 0.938447 0.763932 0.5 0.171573 00.084524 0.5 0.87868 1 1 1 1 1 0.87868 0.5 0.084524 0.307418 0.763932 11 1 1 1 1 1 0.763932 0.307418 0.45049 0.938447 1 1 1 1 1 1 1 0.9384470.45049 0.5 1 1 1 1 1 1 1 1 1 0.5 0.45049 0.938447 1 1 1 1 1 1 10.938447 0.45049 0.307418 0.763932 1 1 1 1 1 1 1 0.763932 0.3074180.084524 0.5 0.87868 1 1 1 1 1 0.87868 0.5 0.084524 0 0.171573 0.50.763932 0.938447 1 0.938447 0.763932 0.5 0.171573 0 0 0 0.0845240.307418 0.45049 0.5 0.45049 0.307418 0.084524 0 0

Also, FIGS. 26 and 27 illustrate example embodiments of weighted circlemasks. In FIGS. 26 and 27, the z axis shows the weight values, and thex-y plane shows the locations of the pixels.

Circular weighted masks may also be used with embodiments that performhistogram-based alignments of images or parts of images (e.g.,subfields). In general, a feature histogram includes the intensityhistogram of the pixels for an area (e.g., a subfield) of an image. Thealignment may be performed by translating the input subfield in alldirections, up to N (e.g., 3) pixels in each direction. The translatedposition that results in the smallest sum of feature differences betweenan input-image subfield and a reference-image subfield may be selectedas the best aligned position. However, if the input-image andreference-image subfields have different rotations, then the differentrotations can cause larger feature differences.

To account for different rotations, some embodiments of theimage-alignment device modify a pixel's contribution to the histogrambin that corresponds to the pixel's intensity based on the mask weightat the pixel's location. For example, a pixel of intensity 1000 with acorresponding mask weight of 0.5 would contribute 0.5 count (the maskweight at the pixel's location) to the histogram bin that corresponds tothe intensity of 1000. And some embodiments of the image-alignmentdevice modify a pixel's intensity according to the mask weight at thepixel's location and then generate a histogram by counting the pixel'scontribution to the histogram bin that corresponds to the modifiedintensity. As an example, a pixel of intensity 1000 with a correspondingmask weight of 0.5 would create a modified intensity of 500 (1000 times0.5), and the modified pixel would contribute one count in the histogramto the bin that corresponds to the intensity of 500.

Histogram gradients may allow the image-alignment device to performoperations, such as subfield-to-subfield alignment, that use a histogramdistance, but in some situations, when creating a histogram using thecircular mask, the continuous nature of the circular mask makes itdifficult to take histogram gradients with respect to the mask center.

For example, consider a dissimilarity measure d(h_(ref), h₂(x_(c),y_(c))) that measures the histogram dissimilarity from a histogramh_(ref) of a reference-image subfield to an input image's subfieldhistogram h₂(x_(c), y_(c)) centered at location x_(c), y_(c). As notedabove, x_(c), y_(c) can be a fractional pixel location. Also forexample, the dissimilarity measure (which measures the histogramdissimilarity) may be an earth mover's distance or a shift-invariantearth mover's distance.

In some embodiments, the histogram is formed using the masking function,and the masking function may take one of the forms that are describedherein. Some embodiments use mask-generated histograms to align the twoimage subfields by minimizing d(h_(ref), h₂(x_(c), y_(c))) over x_(c)and y_(c). Because the histograms generated using a mask generally varysmoothly with sub-pixel shifts, it is possible to minimize the distanceusing techniques such as gradient descent over x_(c) and y_(c) in somerange of possible x_(c) and y_(c)values. The function can also beminimized by considering a grid of values for x_(c) and y_(c).Additionally, a coordinated-descent-type technique may be used that inone step seeks to minimize over x_(c) while keeping y_(c) fixed and thenin a second step fixes x_(c) and minimizes over y_(c). This change ofparameters to minimize over can be repeated until convergence isachieved or for a specified number of iterations.

Some embodiments where the dissimilarity measure is amenable to analysis(such as cosine similarity, for example) attempt to take the gradient ofd(h_(ref), h₂(x_(c), y_(c))) directly and find a solution. In someembodiments, the gradient needs to be approximated or determined throughthe applications of small perturbations in x_(c) and y_(c). tonumerically determine the partial derivatives around some values ofx_(c) and y_(c).

FIG. 28 illustrates an example embodiment of an image-alignment system.The system 10 includes an image-alignment device 2800, which is aspecially-configured computing device; an image-capturing device 2810;and a display device 2820. In this embodiment, the image-alignmentdevice 2800 and the image-capturing device 2810 communicate via one ormore networks 2899, which may include a wired network, a wirelessnetwork, a LAN, a WAN, a MAN, and a PAN. Also, in some embodiments thedevices communicate via other wired or wireless channels.

The image-alignment device 2800 includes one or more processors 2801,one or more I/O components 2802, and storage 2803. Also, the hardwarecomponents of the image-alignment device 2800 communicate via one ormore buses or other electrical connections. Examples of buses include auniversal serial bus (USB), an IEEE 1394 bus, a PCI bus, an AcceleratedGraphics Port (AGP) bus, a Serial AT Attachment (SATA) bus, and a SmallComputer System Interface (SCSI) bus.

The one or more processors 2801 include one or more central processingunits (CPUs), which may include microprocessors (e.g., a single coremicroprocessor, a multi-core microprocessor); one or more graphicsprocessing units (GPUs); one or more tensor processing units (TPUs); oneor more application-specific integrated circuits (ASICs); one or morefield-programmable-gate arrays (FPGAs); one or more digital signalprocessors (DSPs); or other electronic circuitry (e.g., other integratedcircuits). The I/O components 2802 include communication components(e.g., a GPU, a network-interface controller) that communicate with thedisplay device 2820, the network 2899, the image-capturing device 2810,and other input or output devices (not illustrated), which may include akeyboard, a mouse, a printing device, a touch screen, a light pen, anoptical-storage device, a scanner, a microphone, a drive, and acontroller (e.g., a joystick, a control pad).

The storage 2803 includes one or more computer-readable storage media.As used herein, a computer-readable storage medium, in contrast to amere transitory, propagating signal per se, refers to acomputer-readable media that includes an article of manufacture, forexample a magnetic disk (e.g., a floppy disk, a hard disk), an opticaldisc (e.g., a CD, a DVD, a Blu-ray), a magneto-optical disk, magnetictape, and semiconductor memory (e.g., a non-volatile memory card, flashmemory, a solid-state drive, SRAM, DRAM, EPROM, EEPROM). Also, as usedherein, a transitory computer-readable medium refers to a meretransitory, propagating signal per se, and a non-transitorycomputer-readable medium refers to any computer-readable medium that isnot merely a transitory, propagating signal per se. The storage 2803,which may include both ROM and RAM, can store computer-readable data orcomputer-executable instructions.

The image-alignment device 2800 also includes a communication module2803A, a preprocessing module 2803B, a global-alignment module 2803C, anMSS-alignment module 2803D, an anchor-point-selection module 2803E, andan APMSS-alignment module 2803F. A module includes logic,computer-readable data, or computer-executable instructions. In theembodiment shown in FIG. 28, the modules are implemented in software(e.g., Assembly, C, C++, C#, Java, BASIC, Perl, Visual Basic). However,in some embodiments, the modules are implemented in hardware (e.g.,customized circuitry) or, alternatively, a combination of software andhardware. When the modules are implemented, at least in part, insoftware, then the software can be stored in the storage 2803. Also, insome embodiments, the image-alignment device 2800 includes additional orfewer modules, the modules are combined into fewer modules, or themodules are divided into more modules.

The communication module 2803A includes instructions that cause theimage-alignment device 2800 to communicate with one or more otherdevices (e.g., the image-capturing device 2810, the display device2820), for example to obtain one or more images as described in blocksB305 and B330 in FIG. 3 or in blocks B1705 and B1730 in FIG. 17.

The preprocessing module 2803B includes instructions that cause theimage-alignment device 2800 to perform preprocessing on one or moreimages, for example as described in blocks B310 and B335 in FIG. 3 or inblocks B1710 and B1735 in FIG. 17.

The global-alignment module 2803C includes instructions that cause theimage-alignment device 2800 to globally-align two images (e.g., byaligning an image to a reference image) and shift an image based on theresults of the global alignment, for example as described in block B340in FIG. 3 or in block B1740 in FIG. 17.

The MSS-alignment module 2803D includes instructions that cause theimage-alignment device 2800 to split an image into subfields, align eachsubfield with a corresponding subfield in a reference image, and warpeach subfield based on the alignment results, for example as describedin blocks B345-6350 in FIG. 3 or in blocks B1745 to B1750 in FIG. 17.

The anchor-point-selection module 2803E includes instructions that causethe image-alignment device 2800 to select anchor points in an image,select anchor-edge points in an image, and select neighborhood areas inan image, for example as described in blocks B315-6325 in FIG. 3, inblock B1715 in FIG. 17, or in blocks B2100-62185 in FIG. 21.

The APMSS-alignment module 2813F includes instructions that cause theimage-alignment device 2800 to perform APMSS alignment to align an imagewith a reference image and to warp the image based on the results of theAPMSS alignment, for example as described in block B355 in FIG. 3 or inblock B1755 in FIG. 17.

The image-capturing device 2810 includes one or more processors 2811,one or more I/O components 2812, storage 2813, a communication module2813A, and an image-capturing assembly 2814. The image-capturingassembly 2814 includes one or more image sensors and may include one ormore lenses and an aperture. The communication module 2813A includesinstructions that, when executed, or circuits that, when activated,cause the image-capturing device 2810 to capture an image, receive arequest for an image from a requesting device, retrieve a requestedimage from the storage 2813, or send a retrieved image to the requestingdevice (e.g., the image-alignment device 2800).

At least some of the above-described devices, systems, and methods canbe implemented, at least in part, by providing one or morecomputer-readable media that contain computer-executable instructionsfor realizing the above-described operations to one or more computingdevices that are configured to read and execute the computer-executableinstructions. The systems or devices perform the operations of theabove-described embodiments when executing the computer-executableinstructions. Also, an operating system on the one or more systems ordevices may implement at least some of the operations of theabove-described embodiments.

Furthermore, some embodiments use one or more functional units toimplement the above-described devices, systems, and methods. Thefunctional units may be implemented in only hardware (e.g., customizedcircuitry) or in a combination of software and hardware (e.g., amicroprocessor that executes software).

The scope of the claims is not limited to the above-describedembodiments and includes various modifications and equivalentarrangements. Also, as used herein, the conjunction “or” generallyrefers to an inclusive “or,” though “or” may refer to an exclusive “or”if expressly indicated or if the context indicates that the “or” must bean exclusive “or.”

1. A device comprising: one or more computer-readable storage media; andone or more processors in communication with the one or morecomputer-readable media to cause the device to perform operationscomprising: obtaining a reference image; obtaining a test image;globally aligning the test image to the reference image; selectingsubfields in the test image; aligning the subfields in the test imagewith respective areas in the reference image; warping the test imagebased on the aligning of the subfields; selecting anchor points in thereference image; realigning areas of the warped test image withrespective areas in the reference image based on the anchor points inthe reference image; and warping the warped test image based on therealigning of the subfields.
 2. The device of claim 1, wherein aligningthe subfields in the test image with respective areas in the referenceimage generates respective shifts for a plurality of pixels in the testimage, wherein the respective shifts of at least some abutting pixels inthe plurality of pixels are different.
 3. The device of claim 2, whereinwarping the test image includes shifting the plurality of pixelsaccording to the respective shifts.
 4. The device of claim 1, whereinaligning the subfields in the test image with respective areas in thereference image uses cross correlation between the subfields in the testimage and the respective areas in the reference image.
 5. The device ofclaim 4, wherein the cross correlation uses a circular weighted mask. 6.The device of claim 1, wherein the anchor points have respectivecorresponding neighborhood areas, and wherein realigning the areas ofthe warped test image with the respective areas in the reference imageis further based on the neighborhood areas.
 7. The device of claim 1,wherein the anchor points are selected based on intersecting lines inthe reference image.
 8. The device of claim 7, wherein selecting anchorpoints in the reference image includes dividing the respective areas inthe reference image into sub-areas, and identifying an intersection oftwo or more intersecting lines in a sub-area.
 9. The device of claim 8,wherein the operations further comprise: identifying an anchor-edgepoint in the sub-area based on one or more non-intersecting lines in thesub-area, wherein the anchor-edge point is associated with one or moreanchor points in the sub-area.
 10. A method comprising: obtaining areference image; obtaining a test image; globally aligning the testimage to the reference image; selecting subfields in the test image;aligning the subfields in the test image with respective areas in thereference image; warping the test image based on the aligning of thesubfields; selecting anchor points in the reference image; realigningareas of the warped test image with respective areas in the referenceimage based on the anchor points in the reference image; and warping thewarped test image based on the realigning of the subfields.
 11. Themethod of claim 10, wherein aligning the subfields in the test imagewith respective areas in the reference image generates respective shiftsfor a plurality of pixels in the test image, wherein the respectiveshifts of at least some abutting pixels in the plurality of pixels aredifferent.
 12. The method of claim 11, wherein warping the test imageincludes shifting the plurality of pixels according to the respectiveshifts.
 13. The method of claim 10, wherein aligning the subfields inthe test image with respective areas in the reference image uses crosscorrelation between the subfields in the test image and the respectiveareas in the reference image.
 14. The method of claim 13, wherein thecross correlation uses a circular weighted mask.
 15. The method of claim10, wherein the anchor points have respective corresponding neighborhoodareas, and wherein realigning the areas of the warped test image withthe respective areas in the reference image is further based on theneighborhood areas.
 16. One or more computer-readable storage mediastoring computer-executable instructions that, when executed by one ormore computing devices, cause the one or more computing devices toperform operations comprising: obtaining a reference image; obtaining atest image; globally aligning the test image to the reference image;selecting subfields in the test image; aligning the subfields in thetest image with respective areas in the reference image; warping thetest image based on the aligning of the subfields; selecting anchorpoints in the reference image; realigning areas of the warped test imagewith respective areas in the reference image based on the anchor pointsin the reference image; and warping the warped test image based on therealigning of the subfields.
 17. The one or more computer-readablestorage media of claim 16, wherein the anchor points have respectivecorresponding neighborhood areas, and wherein realigning the areas ofthe warped test image with the respective areas in the reference imageis further based on the neighborhood areas.
 18. The one or morecomputer-readable storage media of claim 16, wherein the anchor pointsare selected based on intersecting lines in the reference image.
 19. Theone or more computer-readable storage media of claim 18, whereinselecting anchor points in the reference image includes dividing therespective areas in the reference image into sub-areas, and identifyingan intersection of two or more intersecting lines in a sub-area.
 20. Theone or more computer-readable storage media of claim 19, wherein theoperations further comprise: identifying an anchor-edge point in thesub-area based on one or more non-intersecting lines in the sub-area,wherein the anchor-edge point is associated with one or more anchorpoints in the sub-area.